tt_employment

This notebook is reproduced from the following source: https://www.youtube.com/watch?v=opHDQzhO5Fw

Performning clustering on employment data.

library(dplyr)

Attaching package: 㤼㸱dplyr㤼㸲

The following objects are masked from 㤼㸱package:stats㤼㸲:

    filter, lag

The following objects are masked from 㤼㸱package:base㤼㸲:

    intersect, setdiff, setequal, union

Clean out some NA data and create new variable out of industry and minor_occupation, get the mean of nbr of people in those categories.

employed_tidy <- employed %>%
  filter(!is.na(employ_n)) %>%
  group_by(occupation = paste(industry, minor_occupation),
           race_gender) %>%
  summarise(n = mean(employ_n)) %>%
  ungroup()

`summarise()` regrouping output by 'occupation' (override with `.groups` argument)

Re-shape the data getting it ready for k-means

library(broom)

package 㤼㸱broom㤼㸲 was built under R version 4.0.2

employment_demo <- employed_tidy %>%
  filter(race_gender %in% c("Women", "Black or African American","Asian")) %>%
  pivot_wider(names_from = race_gender, values_from = n, values_fill = 0) %>%
  janitor::clean_names() %>%
  left_join(employed_tidy %>%  # join in total to the pivot wider results
      filter(race_gender =="TOTAL") %>%
      select(-race_gender) %>%
      rename(total = n)) %>%
  filter(total > 1e4) %>% # hard filter to get rid of some observations that have small n
  mutate(across(c(asian, black_or_african_american, women), ~ . / total), # ~ function and . is the variables we have "asian" etc divided by total to create proportions
         total = log(total), # log transform due to the spread in values
         across(is.numeric, ~as.numeric(scale(.)))) %>% # use default scale() the as.numeric(scale(.)) we force it instead of being a matrix to numeric
         mutate(occupation = snakecase::to_snake_case(occupation)) # clean up occupation values into snakecase

Joining, by = "occupation"

For all the different occupation categories to compare how similar these are in terms of these four metrics that are now logged proportions. K-mean is sensitive to how the numbers are centered and scaled.

Which occupations are the most like each other in terms of the demographic representation and in the total (nbr of working in the occupations all together)

Try k-means clustering

summary(employment_clust)

             Length Class  Mode   
cluster      211    -none- numeric
centers       12    -none- numeric
totss          1    -none- numeric
withinss       3    -none- numeric
tot.withinss   1    -none- numeric
betweenss      1    -none- numeric
size           3    -none- numeric
iter           1    -none- numeric
ifault         1    -none- numeric

Use broom to tidy the result of kmeans

These are the three cluster centers in the four dimensional space for each of the dimensions and some additional information.

We can also do additonal things.

We can tell already that there is no clean separation of the clusters. No occupations are cleanly separated from each other. Is clustering the right approach for this? Maybe not.

We can test to at least find best number of clusters.

# check how did the clustering do by its measures in terms of fitting the data
glance(employment_clust)

Choosing k

If we had a data that has strong clusters we would see an elbow here. Tut we do not really see any good elbow point, maybe 5 is the best here but it is not clear.

We redo our clustering

This is an interactive html plot that gives us some more info about the observations. But we see no clear separations, so we can ask how appropriate kmeans is for this kind of data!?